Some notes on the probabilistic semantics of logistic function parameters in neural networks

A common choice for the activation function that is computed at each node in a model neural network is one that is given by the logistie equation , OJ = ,,0+=.=,o,'I'('>:";-<l".=wO,=;,0;=+"";')/"1')" J in whieh OJ is the activation value (in [0,1]) for the jth unit (node), Ij is the s~t {il , ... ,in} of inputs to j, Wlj is the weight (in at) connecting all input i t.o unit j, Zi is the state (usually binary or in [0,11) of i, 8j is the thre1ihold or biM (ill 9?) for j, and T is the tem~ratu~ or r;ain parameter (in Jt+, assumed herein t.o ~ I). The activatioll value oJ is usually thought of A3 represelltinr; either the computed probability of the proposition or event ascribed to j /lS its content, or the probabilit.y thl'.t j, M a binary unit, will tum 011 (i.e., that its state z) will be I rather than 0). The weight and threshold terms are often not given a dear probabilist ic interpretation by thoee .. ho U!iIe them in models, but intuitively a weilht Wij expresses the strength of evidence in favor of (or against) the proposition ~presented by ;(:j that is provided by ;(:1 = I, and the threshold 8j expresses the (positive or negative) t.endency for Zj t.o r;o to I in the ab!Jence of a.ny of the inputs being on or ~true" . If OJ il identified with the probability of &n event or propOliition B j (which could be identital to the event zl = I) given the input state veetor (or vector oftrutb values) x = (;(:i". ,a:,.) then the logit tranlformation of the pl"Qbability P(BJ I x) definet a.n a1t.erna~ive form of ~he logistie equation (with T = I), namely,